AI Toolset—Jupyter Notebook

The terminal icon is in the taskbar at the bottom.

You should see the terminal.

Running Jupyter Lab from the jupyter directory sets the the jupyter directory as the root for all your notebooks, which you can see highlighted in the output. After the server fully loads, it also provides information on how to access the application via a URL, highlighted in the output. Your output may be a bit different compared to the one in the lab guide.

student@student-vm:~$ cd jupyter
student@student-vm:~/jupyter$ jupyter lab
[I 2024-09-19 08:40:33.805 ServerApp] jupyter_ai | extension was successfully linked.
[I 2024-09-19 08:40:33.805 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-09-19 08:40:33.809 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-09-19 08:40:33.813 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-09-19 08:40:33.820 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-09-19 08:40:33.849 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-09-19 08:40:33.850 AiExtension] Configured provider allowlist: None
[I 2024-09-19 08:40:33.850 AiExtension] Configured provider blocklist: None
[I 2024-09-19 08:40:33.850 AiExtension] Configured model allowlist: None
[I 2024-09-19 08:40:33.850 AiExtension] Configured model blocklist: None
[I 2024-09-19 08:40:33.850 AiExtension] Configured model parameters: {}
[I 2024-09-19 08:40:33.940 AiExtension] Registered model provider `ai21`.

<-- Output Omitted -->

[I 2024-09-19 08:40:34.069 ServerApp] Serving notebooks from local directory: /home/student/jupyter
[I 2024-09-19 08:40:34.069 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-09-19 08:40:34.069 ServerApp] http://localhost:8888/lab?token=6f244b92863e2739a45145eddc677c5e99d57b4a9f4f920d
[I 2024-09-19 08:40:34.069 ServerApp]     http://127.0.0.1:8888/lab?token=6f244b92863e2739a45145eddc677c5e99d57b4a9f4f920d
[I 2024-09-19 08:40:34.069 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-09-19 08:40:34.103 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///home/student/.local/share/jupyter/runtime/jpserver-5842-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=6f244b92863e2739a45145eddc677c5e99d57b4a9f4f920d

Notice how the Jupyter Lab landing page is divided in two. On the left side you can see the file and folder explorer, where you can browse and manage your files and folders. On the right, you can see the Launcher tab where you can select to create a Notebook file, or any other supported file format.

Keep in mind that Jupyter Lab sees your /home/student/jupyter folder as its root folder, denoted simply as / within the Jupyter Lab's file explorer.

You should see a new file named Untitled.ipynb appearing in the file explorer on the left. On the right, you should see a new tab labeled with the same filename.

You will see a pop-up window.

The new filename should be visible in the file explorer, and in the editor.

You should see the cursor inside the first cell, marked by a yellow rectangle in the figure.

The default format of a cell is Code. When you select a cell, you can see the format in the main toolbar of the editor.

The format of the cell defines how it is rendered in the Notebook. A code block is meant to contain programming code. You can also set the format of a cell to Markdown or Raw. Markdown is a lightweight markup language that you can use to add formatting elements to plaintext text documents. Raw format is simply meant to display unformatted text. In the following steps, you will use markdown to create a title and descriptions of your code, which is going to be contained within cells set to Code format.

The format of the cell is now set to Markdown.

You should end up with an unrendered cell containing the markdown code you just typed.

You can continue writing markdown text in the cell. Pressing enter will enter a new line in the same block. For a markdown cheat-sheet, visit this website: https://www.markdownguide.org/cheat-sheet/

Notice how the cell still displays the source markdown code.

You will see the rendered text where the title has larger letters in bold. The word Python is formatted in bold, whereas the word Netmiko is in italic. Notice another empty cell appearing right below the formatted text.

In the following steps, you will copy a Python script into the new cell below the rendered text.

You will see the content of the file in the Vi editor.

Before you press the enter key, make sure that you see the command in the bottom-left corner.

After you press the enter key, you should see the text 28 lines yanked into "+ in the bottom-left corner of the editor, which signals a successful copy-to-clipboard operation.

Again, make sure that you see the command :q in the bottom-left corner of the editor. If successful, the terminal with the editor will close.

You will see the cell filled with the color-coded Python code.

The comments are in the lines that start with a # and are colored light blue, and provide a high-level description of what the code does. Notice how part of the code is out of view due to the amount of space available on the display for the editor.

The arrows mark the vertical delimiter between the file explorer and the editor.

Now you should be able to see all of the code and still have the filename visible in the explorer.

You should see the following output below the code:

Connected successfully!
Sending the following command: show ip interface brief

IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status

Your Notebook now contains some markdown-formatted text, a runable code cell with your script, and the output produced by the script.

You deliberately left out a more detailed explanation of the code.

In the file explorer, you will see the Modified column update to Now for the notebook you just saved.

In the following tasks, you will learn how to utilize Jupyter AI features and use a GPT model to explain the code in greater detail and even generate new code that will help you with configuring your devices automatically.

You will see four models. The model name, in the NAME column, is formatted as name:tag. The ID denotes this specific iteration of the model, SIZE is the size on disk, and MODIFIED tells when the model was downloaded or updated. Note, that the order of appearance might differ for you and you might see different timestamps in the MODIFIED column.

student@student-vm:~$ ollama list                         
NAME                ID              SIZE      MODIFIED       
codegemma:latest    0c96700aaada    5.0 GB    3 days ago    
llama3.1:latest     42182419e950    4.7 GB    3 days ago        
mistral:instruct    f974a74358d6    4.1 GB    3 days ago        
phi3.5:latest       61819fb370a3    2.2 GB    3 days ago        

There are many different types of models that are trained on specific tasks. To use the models in a chat-like interface where you want to give them some instructions on what to do, you need models that were specifically trained to follow instructions. All three models are so-called instruct type models. In case of mistral, you can see the tag instruct, letting you know what kind of model it is. The other three models simply have the latest tag. To see additional details, you can visit the Ollama model library. For instance, the phi3.5:latest model properties are in the figure.

In the figure, you can see various parameters - 3.82B, where B stands for billions and the level of quantization - Q4_0 means 4-bit Integer format. Let's shed some light on what the number of parameters and level of quantization actually mean.

GPT models scale based on the number of parameters set during training. Models with more parameters typically have broader knowledge and can solve more complex tasks. The number of parameters determines how much system RAM or GPU memory is required and how long it takes to process a query—larger models need more memory and computation time. GPT parameters are usually stored as 32-bit floating point numbers. To estimate memory requirements in gigabytes (GB), you can check the model's disk usage, as it is roughly the same for inference. Alternatively, you can use the formula M = P × B, where P is the number of parameters (in billions), and B is the number of bytes per parameter. For example, the Phi3.5 model has 3.82 billion parameters. With 32-bit values (4 bytes per parameter), it requires around 15 GB of memory. Note that this number is just a rough estimate, since there is additional data needed for inference, besides the model parameters! For smaller models you can normally increase the number you get with the formula by 20 % to take the additional data into account: M = P x B x 1.20.

Quantization can significantly reduce memory usage and processing time by converting 32-bit values to lower-precision data types, such as 8- or 4-bit integers. This trade-off reduces memory needs at the cost of some loss in answer quality, as fewer bits capture less detail. The Phi3.5 model uses 4-bit integers, which cut the memory requirement down to around 2.2 GB (since 4 bits are half a byte). Aggressive quantization below 16 bits usually reduces answer quality, but llama.cpp allows 4-bit quantization with minimal quality loss. Model size also impacts processing time. For models larger than 13 billion parameters, using a CPU alone is impractical due to long processing times, and a GPU is needed for practical speeds. Note that it applies only to inference—training requires significantly more RAM, typically three to four times as much.

To put things into perspective, ChatGPT 3.5 has around 175B parameters. Without quantization, it would require at least 700 GB of system or GPU RAM for inference and 2100 GB for training.

You should see the chat interface with a message stating additional configuration is required.

You will see the configuration page.

The Language model drop-down list lets you select supported AI providers and GPT models that are used for regular inferencing tasks. These models will process your queries and generate answers. The Embedding model drop-down list contains various embedding models supported by the AI providers. These models are specially trained and designed to convert text into numeric vectors, a core component of RAG applications. The Inline completions model can be set to suggest code snippets in real time during development. It takes into account the entire Notebook as the history from which it derives code completion suggestions. Below you can also set the API keys needed for cloud AI providers, such as OpenAI (ChatGPT), Hugging Face Hub, and so on. You can also set whether pressing the enter key in the chat interface creates a new line or sends the message. Keep in mind that the default behavior is to send the message to the GPT. For this task, you will configure only the core Language model for regular inferencing tasks. Since you will be using an on-prem inferencing server, you will not use any API keys.

You will see a list of available AI providers and GPT models in the provider::model notation.

Notice how there is an asterisk after the double colon. The notation provider::* is reserved for provider::model combinations that are not hardcoded in advanced in Jupyter AI. You specify the exact model ID in the following step in these cases. You can also see that the Hugging Face Hub has the same notation.

Notice how the Completion model is populated with the Ollama::* entry. You can also see the additional fields for Local model ID and Base API URL. The Local model ID is the name of the GPT model that you want to use. The Base API URL is the URL of the Ollama inference server that you want to use. Normally, you have to specify the URL if the Ollama server is running on a different machine or is using a nondefault port. The Ollama server is running on the same machine as Jupyter AI and was installed with the default settings, so you can leave the URL field empty and let Jupyter AI use its default settings.

The Save Changes button is in the bottom-left corner of the settings page.

You will see a green message notifying you that the settings were saved successfully.

You will see the introduction text of Jupyternaut - the chatbot, and some useful tips.

The /ask and /learn commands pertain to the built-in Retrieval-Augmented Generation (RAG) pipeline, which is an advanced feature, enabling file lookup capabilities, and will not be covered in this exercise. Let's test the selected model and see a couple of these commands in action.

Your chat interface should look like the one in the figure.

After around 20s, the answer will start streaming. Note that GPT models are statistical by nature, which means that there is no idempotency; entering the same prompt will not produce the exact same answer! From here on in, the answers you will get are probably going to be significantly different to the ones provided in the lab guide.

Based on the answer we got, you can see that Jupyter AI correctly recognized phi3.5:latest as the selected GPT model. It also provided an additional description about Jupyternaut's capabilities. You can see from the answer that Jupyternaut is not a regular chatbot, but a complex application based on large language models (LLMs). Jupyternauts answers can be quite verbose and take up a lot of screen space. Let's clear the current chat.

You will see a drop-down menu with available commands, that were also described at the welcome page a couple of steps before.

The chat window should now be clear and only the command summary page should be visible below the prompt field.

The integration of AI in Jupyter Lab works as expected. This type of chatbot usage is fairly common, but the real strength of Jupyter AI lies in how AI is integrated in the Notebook itself. In the following task, you will explore an additional integration of AI within the Jupyter Lab workflow, such as code explanation, generation, and debugging.

The folder icon is located at the top of the left navigation panel.

You should see the NetworkAutomation.ipynb file in the root folder.

You should see the notebook open on the right. If you have the launcher tab present, the notebook will open next to it. You can close the launcher tab. Without the launcher tab, your screen should now look like in the figure.

It is located at the bottom of the left navigation panel.

You will see the command summary page.

You should see blue vertical bars next to the cells with the code and its output as well as a blue rectangle around the cell with the code.

You will see a drop-down menu.

You will see another drop-down menu with the options to Explain, Fix, Optimize, and Refactor code.

The Fix code option is grayed out since this works only if you select a code cell that produced an output with some error message.

You will see that Jupyter AI automatically generates a prompt with the instruction to explain the code and the code itself copied from the cell. It might take around 20 seconds for the GPT to start generating the response.

We got a long and verbose response. For clarity, it will be presented in multiple parts. You can see that the answer starts with a nonsensical phrase "Certain extranet!" This type of wrong response is normally defined as hallucination. Hallucination is more profound with small models since their knowledge base is quite limited. Next, it explains the line where you import the Netmiko library. The explanation continues with explaining the device parameters. The explanation of the parameters is quite good, except the part where it tries to explain the device_type parameter where the switch OS is defined. This explanation is offered inside a code snippet in the form of python comment lines. The explanation of the try-except routine is also on point.

The second part of the response begins with the explanation of the code responsible for sending the CLI command and retrieving the output from the switch. It ends by discussing the disconnection routine and why it is a good practice.

The small Phi3.5 with only 3 billion parameters was capable of explaining what the code does, despite some trouble with reasoning when it came to the greeting and Cisco OS definitions. All things considered, the response was factually correct, and easy to understand. You can use this approach whenever you come across a script you would like to use, but don't really understand what it does. It makes it easy to use existing code that you can augment to meet your needs.

You probably noticed that the model is quite verbose and uses a relaxed, friendly, and optimistic tone. You can instruct the GPT model to write in a certain tone and level of verbosity by recreating the prompt with these instructions. For example, you could write a prompt like this: "Explain the code. Use concise and professional language. The code: <paste the code here>". Feel free to try different instructions and experiment with the model to learn how to make it respond according to your wish.

Now let's see whether a model with only 3 billion parameters is capable of writing code based on instructions.

You should see the command summary page again.

You should see the following prompt:

Write a simple Python script that will configure the ethernet 1/1 interface IP address to 172.16.0.111/24 on a Cisco Nexus v9300 switch. You can only use the Netmiko library. The switch can be accessed by SSH on 172.16.0.10. Do not provide explanations of the code.

It will take around 20 seconds for the GPT to start writing the response. Your response will most likely be very different.

At a first glance, the script we got looks good. One glaring issue is the wrong netmask set to 255.255.255.128 instead of 255.255.255.0. In general, GPT models tend to make mistakes with precise data, such as converting a /24 CIDR notation to the regular octet format, so you always have to check and verify all the answers GPT models provide, regardless of how big the model you are using is! This kind of prompting, where you describe a task without much detail, is called one-shot prompting and is normally enough for simple tasks. When doing more complicated tasks, a better prompt with some examples is necessary - a technique called few-shot prompting. Before moving on, you will test the script with the proper credentials and the correct netmask.

You will see a Copied! notification right above the icon.

The Notebook should now contain the pasted code as well.

The fixed parts of the code are highlighted in the figure.

You will see the prompt asking for the password from the getpass() function call. If you are using your own response, you might not have this function call in your script. If so, skip the next step.

You will see the output of the code below.

The code failed. These error messages are often very complicated and hard to understand, especially if you are not well versed in programming. You can use Jupyter AI to help you explain what went wrong and how to fix it.

The GPT model created a prompt based on the /fix command and the active cell.

After a minute or two, we got the following explanation of the error message.

The following figure shows the list of supported devices. Notice that you must use cisco_nxos instead of cisco_nexus for the device_type.

Let's fix the error and see if the script now works.

Notice how the blue bars changed to orange, indicating a change in the code.

You will got the following output from the switch.

Notice how the output now changed to show the configured IP address on the interface. Notice also that it is currently shut down, an issue the script did not address because it lacked the no shutdown command.

If you were using your own response, you can follow the same troubleshooting procedure.

Your notebook should look like the one in the figure. Note, that the notebook tab is extended for better visual clarity.

The %load_ext jupyter_ai_magics command loads the necessary IPython extension, needed by the %%ai command.

The command produces no output.

This is the content in the /home/student/prompts/prompt2.txt file:

%%ai ollama:codegemma --format code
Write a Python script using netmiko to configure a Cisco Nexus device. The script should connect to the device with the IP 172.16.0.10, using the username admin and password C!sco123.
The script should create VLAN 10, configure the interface Ethernet1/2 as a Layer 2 switchport, set it as an access port assigned to VLAN 10, and enable the interface.
The script should show the output after each step.

Your notebook should look like the following.

Let's examine the command %%ai ollama:codegemma --format code. The %%ai is a cell magic command, meaning it applies to the entire cell, including the prompt text that follows. The syntax ollama:codegemma specifies the AI provider and the model to be used, in this case, the Ollama inference server running the codegemma model. This model is specifically designed to assist with coding tasks, offering improved capabilities in code generation, explanation, and debugging compared to previously used models. By default, Jupyter AI assumes that the output from a model will be in markdown format. However, using the --format parameter can change this default setting. Here, --format code ensures that the output is in Python syntax. Other supported formats include text, json, html, and math.

The prompt in this example is crafted with precision. GPT models are capable of solving tasks of varying complexity, but their effectiveness can be enhanced by reducing complex tasks into smaller problems. This process is known as prompt engineering. In this case, the first sentence of the prompt specifies the programming language and operating system. It then outlines how the script connects to the device. Next, the prompt divides a complex task into detailed step-by-step instructions. Even the most advanced GPT models require carefully constructed prompts to deliver high-quality responses. Generative AI, therefore, serves as a tool. In the hands of a skilled network engineer, it can augment the user's capabilities, but it cannot replace the engineer, as it depends on the engineer to provide meaningful instructions to the GPT model.

You can see the output that we got in the following figure.

After reviewing our code, you notice that almost everything is in order. This script begins by connecting to the device, and it continues by configuring a VLAN with the ID 10 and showing the output of the show vlan brief command as verification. Next, it starts configuring the Ethernet1/2 interface. However, Nexus v9300 switches default to having all interfaces in Layer 3 mode. To configure an interface as a Layer 2 access port and assign it to a VLAN, you must first enter the switchport command, which is absent in the script provided by the AI. The rest of the configuration is correct. Also, while the show interface status command displays the status of all interfaces, you only need to verify the status of the Ethernet1/2 interface. Case in point, generative AI can save you time, but it cannot take the place of an experienced network engineer's knowledge and skills. Generative AI can and will make mistakes, so it's important to always check its output. Use your knowledge and experience to spot and fix any errors it might have made.

You should see the code inside the Vi editor.

The command should be visible in the bottom-left corner of the terminal window.

You should see the following message at the bottom of the terminal window: 28 lines yanked into "+

Your notebook should look similar to the one in the figure.

The changed lines are highlighted in the figure.

If you followed the example in the lab guide, you will get the following expected output.

If you used your own code and encountered any errors, simply right-click in the cell with the code, select Generative AI and click the Fix code option and see if codegemma can help you fix the error.

Using generative AI effectively involves a lot of trial and error. You are encouraged to experiment with different GPT models to learn how to create effective prompts and determine which models best meet your needs. Keep in mind that general GPT models perform adequately across many tasks but may not excel. Specialized GPT models are better suited for specific tasks. Experiment with various models to find the model that fits your requirements. For coding tasks, the CodeGemma models are a good starting point. For general text generation and simpler coding tasks, models like Llama3.1, Mistral, and Phi3.5 are effective and efficient choices.